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Purpose of Presentation 


Adapt / combine known methods to demonstrate an expert 
judgment elicitation process that ... 

1 . Models expert’s inputs as a triangular distribution 

- 12 questions to elicit required parameters for a bounded distribution 

- Not too complex to be impractical; not too simple to be too subjective 

2. Incorporates techniques to account for expert bias 

- A repeatable Q&A process that is iterative & includes visual aids 

- Convert Triangular to Beta-PERT (if overconfidence was addressed) 

3. Is structured in a way to help justify expert’s inputs 

- Expert must provide rationale for each of his/her responses 

- Using Risk Breakdown Structure, expert specifies each risk factor’s relative 
contribution to a given uncertainty (of cost, duration, reqt, etc.) 

This paper will show one way of “extracting” expert opinion for 
estimating purposes . Nevertheless , as with most subjective 
methods , there are many ways to do this . 
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The Uncertainty Spectrum 


No Estimate Required 

Total Certainty = Complete information 


All known 


Specific Uncertainty 


Partial information 


General Uncertainty 


Total Uncertainty = No information 


Objective 

Probabilities 


V 


A 


Subjective 

Probabilities 


V 


Data / 
Knowledge 


Known unknowns 



Unknown unknowns 


No Estimate Possible 


Reference: Project Management Consulting by AEW Services, 2001 


Expert opinion is useful when little information is available for 
system requirements, system characteristics, durations & cost 
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Expert Judgment Elicitation (EE) 


EXPERT JUDGEMENT ELICITATION PROCEDURE 


STRUCTURED APPROACH TO CAPTURING AN 
EXPERTS KNOWLEDGE BASE AND CONVERT HIS\HER 
KNOWLEDGE BASE INTO QUANTITATIVE ASSESSMENTS. 


MODELERS SKILLED IN 
DECOMPOSITION 
AND AGGREGATION OF 
ASSESSMENTS 

I' 

I 

I 


EXPERTS 



NORMATIVE 



SUBSTANTIVE 

l 

l 


J KNOWLEDGABLE ABOUT 

j THE SUBJECT MATTER 

I AND EXTENSIVE EXPERIENCE 


ELICITATION PROCESS = 
MULTIPLE CYCLES (AT LEAST 2) 


1. DECOMPOSITION OF 
EVENT OF INTEREST TO 
A MEANINGFULL LEVEL FOR 
SUBSTANTIVE EXPERT 


2. ELICITATION OF JUDGMENT OF 
SUBSTANTIVE EXPERT FACILI- 
TATED BY NORMATIVE EXPERT 


3. AGGREGATION OF 
JUDGEMENTS 
BY NORMATIVE EXPERT 


Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen 
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Triangular Distribution 


• Used in situations were there is little or no data 


- Just requires the lowest (L), highest (H) and most likely values (M) 


Each x-value has a respective f(x), sometimes called 
“Intensity” that forms the following PDF : 

2 (x-L) 


/(*) = 


L< x < M 


, M <x< H 


(M —L)(H — L) 
2(H —x) 

~ ( H-M)(H-L ) 

= 0, otherwise 


L, M & H are all that ’s needed to calculate the 
Mean and Standard Deviation: 




c L + M + H ) 


a = 


\L 2 +M 2 + H 2 -L*M-L*H-M*H) 


18 
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Beta Distribution 


Bounded on [0,1] interval, scale to any interval & very flexible shape 


/(*) = 


f i Yr(a+py 




\H-Lj 
= 0 otherwise 


(a)T(fmH-L 


J 


H-x 

\H-Lj 


\M 


L<x<H Shape Parameters :a>0, p>0 


Calculated 
Gamma values 
using Excel’s 
GAMMALN 
function: 


T(a + p) = EXP[GAMM AIN(a + p)] 
T(a) = EXP[GAMM AIN(a)] 
T(p) = EXP[GAMM AIN(p)] 



Most schedule or cost estimates follow right skewed pattern. But 
how do we know a and p? Answer: Beta-PERT Distribution. 


Sources: 1 . Dr. Paul Garvey, Probability Methods for Cost Uncertainty Analysis, 2000 
2. LaserLight Networks, Inc, “Beta Modeled PERT Schedules” 
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Beta-PERT Distribution 


Requires lowest (L), highest ( H ) & most likely values (M) 


UseL, M and H to ( L + X*M+H ) 

calculate mean(ju) and ju = 

standard deviation (<j) : X + 2 


a = 1-lid. 
6 


t/se L, H, ju and cr 
To calculate shape 
parameters, a & j3 : 


r (y-D „ (n-L)(7/-n) | 

( H-L ) ct 2 



(fj-L) 


* a 


where a > 0, p > 0 


a aft J 1 3 are needed to define the Beta Function and compute the Beta Probability Density: 


Beta Probability 
Density Function 
(as shown in slide 9): 


/(*)= 


( i ^ 



Calculated Gamma 
values using Excel’s 
GAMMALN function: 


x-L 

H-Lj 




H-x 

\U-Lj 


-1 


L<x< H 


* 


r(a + P) = EXP[GAM M ALh(a + p)] 
T(a) - EXP[GAMM AL^a)] 
T(p) = EXP[G AM M ALhfP)] 


Sources: 1 . Dr. Paul Garvey, Probability Methods for Cost Uncertainty Analysis, 2000 
2. LaserLight Networks, Inc, “Beta Modeled PERT Schedules” 
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Expert Elicitation (EE) Phases 


Expert Elicitation consists of five phases: 

(note that Phases 4 & 5 are iterative) 

I . Motivating the expert 
; ► 2 . Training (conditioning) the expert 
I *3. Structuring objective, assumptions & process 
—4. Assessing (encoding) expert’s responses — 

• Q&A - Expert’s technical opinion is elicited 

• Quantitative results w/ documented rationale 

— 5. Verifying encoded values & documentation 

Our Example will emphasize the Phase 4 Q&A 
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Example: Estimate Commute Time 


• Why this example? 

- Fairly easy to find a subject matter expert 

- It is a parameter that is measurable 

- Most experts can estimate a most likely time 

- Factors that drive uncertainty can be readily identified 

- People general care about their morning commute time! 

Let's begin with Phase I ... Motivating the Expert: 

I . Motivating the expert 

• Explain the importance & reasons for collecting the data 

• Explore stake in decision & potential for motivational bias 
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EE Phase 2: Commute Time 


2 . Structuring objective, assumptions & process 

• Be explicit about what you want to know & why you need to know it 

Clearly define variable & avoid ambiguity and explain data values that are required 
(e.g. hours, dollars, %, etc) 

The Interviewer should have worked with you to develop the 
Objective and up to 5 Major Assumptions in the table below 

• Please resolve any questions or concerns about the Objective and/or 
Major Assumptions prior to continuing to "Instructions". 

Objective : Develop uncertainty distribution associated with time (minutes) 
it will take for your morning commute starting 1 October 2014. 

Assumption 1 : Your commute estimate includes only MORNING driving time 
Assumption 2: The commute will be analogous to the one you've been doing 
Assumption 3 Period of commute will be from 1 Oct 2014 thru 30 Sep 2015 
Assumption 4 Do not try to account for extremely rare & unusual scenarios 
Assumption 5: Unless you prefer otherwise, time will be measured in minutes 
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EE Phase 3: Commute Time 


3. Training (conditioning) the expert 

• Go over instructions for Q&A process 
9 Emphasize benefits of time constraints & 2 iterations 


Instructions : This interview is intended to be conducted in two Iterations. 

Each iteration should take no longer than 30 minutes. 

A. Based on your experience, answer the 12 question sets below. 

B. Once you've completed the questions, review them & take a 15 minute break. 

C. Using the triangular graphic to assist you, answer all of the questions again. 

Notes: 

A. The 2 nd iteration is intended to be a refinement of your 1 st round answers. 

B. Use lessons-learned from the 1 st iteration to assist you in the 2 nd iteration. 

C. Your interviewer is here to assist you at any point in the interview process. 
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EE Phase 3: Commute Time (cont’d) 


3. Training the expert (continued) 

For 2 Questions, you’ll need to provide your assessment of likelihood: 


Descriptor 

Explanation 

Probability 

Absolutely Impossible 

No possibility of occurrence 


Extremely Unlikely 

Nearly impossible to occur; very rare 


Very Unlikely 

Highly unlikely to occur; not common 


Indifferent between "Very Unlikely" & "Even chance" 

Values 
will be 
defined 

Even Chance 

50/50 chance of being higher or lower 

Indifferent between "Very Likely" & "Even chance” 

by SME 

Very Likely 

Highly likely to occur; common occurrence 


Extremely Likely 

Nearly certain to occur; near 100% confidence j 


Absolutely Certain 

100% Likelihood 



Example: Assume you estimated a "LOWEST" commute time of 20 minutes. 

Your place a value = 1 0.0% as the probability associated with "Very Unlikely." 

Therefore: 

a) You believe it's "VERY UNLIKELY" your commute time will be less than 20 minutes, and 

b) This is equal to a 1 0.0% chance that your commute time would be less than 20 min. 
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EE Phase 4: Commute Time (iteration I ) 

Question I: Expert creates “value-scale” tailored his/her bias ... 

What probability would you assign to a value that's "Very Unlikely" 

What probability would you assign to a value that's "Extremely Unlikely" 


Available Selection ofValues to the Expert (shaded cells were selected by expert): 
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EE Phase 4: Commute Time (iteration I ) 


Question I: Expert creates “value-scale” tailored his/her bias ... 

What probability would you assign to a value that's " Very Unlikely " 
What probability would you assign to a value that's " Extremely Unlikely'' 


Descriptor 

Explanation 

Probability 

Absolutely Impossible 

No possibility of occurrence 

0.0% 

Extremely Unlikely 

Nearly impossible to occur; very rare i 


1 .0% 

Very Unlikely 

Highly unlikely to occur; not common 1 


10.0% 

Indifferent between "Very Unlikely" & "Even chance" 

30.0% 

Even Chance 

50/50 chance of being higher or lower 

r 

50.0% 

Indifferent between "Very Likely" & "Even chance" 

70.0 % 

Very Likely 

Highly likely to occur; common occurrence 

90.0% 

Extremely Likely 

Nearly certain to occur; near 100% confidence 

99.0% 

Absolutely Certain 

100% Likelihood 

100.0% 


Only 2 probabilities needed to be elicited in order to 
create aValue-Scale that has 9 categories! 
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EE Phase 4: Commute Time (Iteration I ) 


4. Assessing expert’s responses (Q&A) 

Given the objective and assumptions ... 

2. Describe input parameter (WBS 4): Commute (in minutes) as Triangular Dist’n 


3. What’s the Most Likely value, M? 50 

4. Adjust M (if applicable) 55 

5. What’s the chance actual value could exceed M? N/A 

6. What’s the Lowest value, L 42 

7. What’s the chance actual value could be less than L? Indifferent-Low 

8. What’s the Highest value, H 80 


9. What’s the chance actual value could be higher than H? Very Unlikely 


This I st iteration tends to result in anchoring bias on M, 
over-confidence on L and H, and poor rationale 
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EE Phase 4: Commute Time (iteration I ) 


4 . Assessing expert’s responses (Q&A) 


User-Provided Distribution for Commute Time 

Red dot depicts unadjusted point estimate. Dashed lines depict unadjusted lowest & highest 



L H 


PDF created 
based upon 
Expert's 
responses to 
Questions 2 
through 9. 


Given from Expert : L=42, M=55, H=80, p(x<L)=0.30 and p(x>H)=0. 1 0 
Calculation of ‘true* L and H ( a ) : L = 1.56 and H = 101.15 ... Do these #'s appear reasonable? 


(a) Method to solve for L and H presented in “Beyond Beta,” Chi (The Triangular Distribution) 
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EE Phase 4: Commute Time (iteration I ) 


Question 10: Expert & Interviewer brainstorm risk factors ... 
What risk factors contributed to the uncertainty in your estimate ? 


Weather 

Accident(s) 

Road Construction 

Departure Time 

Red Lights 

Emergency vehicles 

School buses 

Not feeling well 

Inexperienced driver 
Unfamiliar with route 


Create Risk 
Breakdown 
Structure (RBS) 



Objective Means Barriers / Risks 


Maximize 
Average Speed 



Weather 

Avoid 

Accident(s) 

Dense Traffic 

Road Construction 


Departure Time 


Red Lights 

Avoid stops 

Emergency vehicles 


School buses 


Not feeling well 

Optimize driving 

Inexperienced driver 


Unfamiliar with route 


Question I I: Expert selects top 6 risk factors ... 

What are the top 6 risk factors that contributed to your estimate uncertainty? 

User Input Examples or Justification: 


Weather 

Rain, snow & especially ice, have caused major delays in the past; 1 expect similar impacts in 2014. 

Accident(s) 

Accidents occasionally occur. In some cases, these have added 60 minutes to my commute! 

Road Construction 

Sometimes road crews shutdown 1 or 2 lanes; typically adding 10-20 minutes to my commute. 

Departure Time 

Itry to leave 1 hour before rush hour. Leaving later can add 10-15 minutes to my commute. 

Not Feeling Well 

If I'm not feeling w ell, I'll drive more slow ly or even make a w rong turn! Can add 5 min to commute. 

Red Lights 

1 tend to "catch" the same lights every day so this factor could add 1-2 minutes to my commute. 
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EE Phase 4: Commute Time (iteration I ) 

Question 12: Expert scores each risk factor’s contribution to uncertainty ... 
Score each risk factor a value based upon the following instruction: 


If the specified risk factor: * 

is the largest contributor to uncertainty (e.g. biggest driver of H) 

then score it a 

5.0 

Indifference 


4.5 

is a significant contributor to uncertainty (e.g. big driver of H) 

then score it a 

4.0 

Indifference 


3.5 

has a moderate effect on uncertainty (e.g. nominal impact on H) 

then score it a 

3.0 

Indifference 


2.5 

has a small effect on uncertainty (e.g. not a big driver of H) 

then score it a 

2.0 

Indifference 


1.5 

is the smallest contributor to uncertainty (e.g. smallest driver of H) 

then score it a 

1.0 


* Note: You can have 2 or more risk factors with a score of 5 (or score of 1). 

~l Risk Factor Score 


Expert provides 
a score for each 
risk factor 
(rationale not 
shown). 


The 1st iteration of Q&A is complete. Recommend the 
expert take a 1 5 minute break before re-starting Q&A 
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Weather 

5.0 

Accident(s) 

5.0 

Road Construction 

2.0 

Departure Time 

4.0 

Not Feeling Well 

1.0 

Red Lights 

1.5 




EE Phase 4: Commute Time (Iteration 2) 


4. Assessing expert’s responses (Q&A) 


Given the objective, assumptions & input parameter (WBS4): 


1. Do you need to modify the probability value scale? No 

2. Do you need to re-characterize the input parameter? No 

3. Do you want to adjust your Most Likely Value, M? No 

4. What’s the chance the actual value could exceed M? N/A 

Assuming best case: weather, accidents, road const, departure time, etc . Document Scenario 

5. What’s the Lowest value, L 40 

6. What’s the chance actual value could be less than L? Extremely Unlikely 
Assuming worst case: weather, accidents, road const, departure time, etc. Document Scenario 

7. What’s the Highest value, H 90 


8. What’s the chance actual value could be higher than H? Indifferent-Low 
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EE Phase 4: Commute Time (iteration 2) 


4 . Assessing expert’s responses (Q&A) 

User-Provided Distribution for Commute Time 

Red dot depicts unadjusted point estimate. Dashed lines depict unadjusted lowest & highest 



PDF created 
based upon 
Expert’s 
responses to 
Questions I 
through 8. 


Given from Expert : L-40, A 1=55, H=90, p(x<L)=0. 1 0 and p(x>H)=0.30 


Calculation of ‘true* L and H ( a ) : L = 35.44 and H = 143.92 ... Do these #’s appear reasonable? 

2 nd iteration helps “condition” expert to reduce anchoring bias on M, 
counter over-confidence on L & H, calibrate Values’ & improve rationale. 
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EE Phase 5: Commute Time (iteration 2) 


5. Verifying encoded values & documentation 


Triangular PDF from Iteration I 


Triangular PDF from Iteration 2 


User-Provided Distribution for Commute Time 

User-PrOVided Distribution for Commute Time Red dot depicts unadjusted point estimate. Dashed lines depict unadjusted lowest & highest values 

Red dot depicts unadjusted point estimate. Dashed lines depict unadjusted lowest & highest 



Inputs not necessarily sensitive to 
risk factors => Optimistic Bias 


Inputs sensitive to weighted risk 
factors => Minimum-Bias 


The 2 nd iteration helped elicit an L that seems feasible 
and an H that accounts for worst-case risk factors 
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Results (Triangular & Beta-PERT) 



Commute Time (minutes) 

• In most cases, Beta-PERT is preferred (vs triangular) 

- Beta-PERT’s mean is only slightly greater than its mode 

• However, triangular would be preferred (vs Beta-PERT) if elicited 
data seems to depict over-confidence (e.g. H value is optimistic) 

- Triangular PDF compensates for this by ‘exaggerating’ the mean value 
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Conclusion 


We provided an expert elicitation overview that ... 

1 . Demonstrated a way to model expert opinion as a 
triangular distribution 

- A process that does not “over-burden” the subject matter expert 

2. Incorporated techniques to address expert bias 

- Iterative Q&A process that includes use of visual aids 

- Relied on at least a 2 nd iteration to help minimize inaccuracy & bias 

- Convert Triangular to Beta-PERT (if overconfidence was addressed) 

3. Structured the process to help justify expert’s inputs 

- Rationale required for each response 

- RBS to help identify what risk factors contribute to uncertainty 

- Weight risk factors to gain insight as each risk factor’s relative 
contribution to uncertainty (cost, schedule, etc.,) 
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Potential Improvements 


• More upfront work on “training” the expert 

• Criteria when to elicit mean or median (vs mode) 

• Add 2 questions to create Modified Beta-PERT 

• Improve scaling tables that depict expert’s judgment 

• Create “starter” Risk Breakdown Structures” 

- Facilitates brainstorming process of possible risk factors 

• Convert best case & worst case scenarios to probabilities 

• Improve method of weighting risk factors 

• Explore other distributions, e.g. Weibull & LogNormal 

• Incorporate methods to combine expert judgments 

So ... hopefully ... this adds to the conversation on how 
best to leverage expert judgment in the cost community. 
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intuition versus Analysis 


Quickly answer the question: 


**A bat and a ball cost $ 1.10 in total. 
The bat costs $ I more than the ball. 
How much does the ball cost?.” 
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Sources not Referenced in Presentation 


1 . Liu, Y., “Subjective Probability,’’ Wright State University. 

2. Kirkeboen, G., “Decision behaviour - Improving expert judgement, 20 1 0 

3. Vose, D., Risk Analysis (2 nd Edition), John Wiley and Sons, 2004 

4. “Expert Elicitation Task Force White Paper,’’ US EPA, 2009 

5. Clemen, R.T. and Winkler, R.L. (1990) Unanimity and compromise among 
probability forecasters. Management Science 36 767-779 
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A Step-Wise Approach to Elicit 
Triangular Distributions 


Formerly entitled “An Elicitation Method to Generate 
Minimum-Bias Probability Distributions” 

Questions? 


Marc Greenberg 
202.358.1025 

marc.w.greenberg@nasa.gov 
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Probability Distributions 


Bounded 

• Triangular & Uniform 

• Histogram 

• Discrete & Cumulative 

• Beta & Beta-PERT 

Unbounded 

• Normal & Student-t 

• Logistic 

Left bounded 

• Lognormal 

• Weibull & Gamma 

• Exponential 

• Chi-square 


Non-Parametric Distributions: Mathematics 
defined by the shape that is required. 
Empirical, intuitive and easy to understand. 


Parametric Distributions: Shape is born of 
the mathematics describing theoretical 
problem. Model-based. Not usually intuitive. 


Of the many probability distributions out there, Triangular & Beta- 
PERT are among the most popular used for expert elicitation 
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Reasons For <& Against Conducting EE 

Reasons for Conducting an Expert Elicitation 

• The problem is complex and more technical than political 

• Adequate data (of suitable quality and relevance) are unavailable or unobtainable in the 
decision time framework 

• Reliable evidence or legitimate models are in conflict 

• Qualified experts are available & EE can be completed within decision timeframe 

• Finances and expertise are sufficient to conduct a robust & defensible EE 

Reasons Against Conducting and Expert Elicitation 

• The problem is more political than technical 

• A large body of empirical data exists with a high degree of consensus 

• Findings of an EE will not be considered legitimate or acceptable by stakeholders 

• Information that EE could provide is not critical to the assessment or decision 

• Cost of obtaining EE info is not commensurate with its value in decision-making 

• Finances and/or expertise are insufficient to conduct a robust & defensible EE 

• Other acceptable methods or approaches are available for obtaining the needed 
information that are less intensive and expensive 
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Some Common Cognitive Biases 


• Availability 

- Base judgments on outcomes that are more easily remembered 

• Representativeness 

- Base judgments on similar yet limited data and experience. Not fully 
considering other relevant, accessible and/or newer evidence 

• Anchoring and adjustment 

- Fixate on particular value in a range and making insufficient adjustments 
away from it in constructing an uncertainty estimate 

• Overconfidence (sometimes referred to as Optimistic bias) 

- Strong tendency to be more certain about one’s judgments and 
conclusions than one has reason. Tends to produce optimistic bias. 

• Control (or “Illusion of Control”) 

- SME believes he/she can control or had control over outcomes related to 
an issue at hand; tendency of people to act as if they can influence a 
situation over which they actually have no control. 
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